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The University of Arkansas 

was founded in 1871 as the flagship institution 
of higher education for the state of Arkansas. 
Established as a land grant university, its mandate was threefold: to teach students, conduct 
research, and perform service and outreach. 

The College of Education and Health Professions established the Department of Education 
Reform in 2005. The department’s mission is to advance education and economic development 
by focusing on the improvement of academic achievement in elementary and secondary schools. 
It conducts research and demonstration projects in five primary areas of reform: teacher quality, 
leadership, policy, accountability, and school choice. 

The School Choice Demonstration Project (SCDP), based within the Department of Education 
Reform, is an education research center devoted to the non-partisan study of the effects of school 
choice policy and is staffed by leading school choice researchers and scholars. Led by Dr. Patrick 
J. Wolf, Professor of Education Reform and Endowed 21st Century Chair in School Choice, 
SCDP’s national team of researchers, institutional research partners and staff are devoted to the 
rigorous evaluation of school choice programs and other school improvement efforts across the 
country. The SCDP is committed to raising and advancing the public’s understanding of the 
strengths and limitations of school choice policies and programs by conducting comprehensive 
research on what happens to students, families, schools and communities when more parents are 
allowed to choose their child’s school. 
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Executive Summary 

With the passage of 2005 Wisconsin Act 125, private schools participating in the Milwaukee Parental 
Choice Program (MPCP) are now required to administer a nationally normed standardized test annually 
in reading, mathematics, and science to their MPCP (a.k.a. “Choice”) students enrolled in the 4 th , 8 th , 
and 10 th grades. The law further directs Choice schools to submit copies of the scores from those tests to 
the School Choice Demonstration Project (SCDP) for processing and reporting to the Legislative Audit 
Bureau. During the 2006-07 school year, MPCP schools administered either nationally normed tests, 
such as the Iowa Test of Basic Skills, or the criterion referenced Wisconsin Knowledge and Concepts 
Examinations (WKCE). The School Choice Demonstration Project received 5,194 nationally normed 
scores from 66 schools and 1,231 WKCE scores from 40 schools. 




The average nationally normed test scores received by the SCDP from the Choice schools ranged from 
the 28 th to near the 39 th percentiles, depending on the grade level and subject area. Such relatively low 
levels of performance are typical 
for the population of low-income 
urban students served by the MPCP. 

Fourth grade students in the MPCP 
on average scored around or below 
the lower-third of the performance 
distribution in all three subject 
areas. The Choice students tested in 
8 th grade performed slightly better, 
but still near the 33 rd percentile. 

The MPCP students tested in 
10 th grade, on average, performed 
somewhat higher than their 8 th and 
4 th grade counterparts, from the 34 th 
percentile in math to near the 39 th 
percentile in reading and science. 



The performance of the Choice 

students who took the criterion referenced WKCE can be compared to that of similarly income- 
disadvantaged students in Milwaukee Public Schools (MPS) who took the same test. The 4 th grade 
MPCP students who took the WKCE on average scored 8 to 13 scale score points (equal to .16 to .29 of 
a standard deviation) below the average scores of income-disadvantaged 4 th graders in MPS. The 8 th grade 
MPCP students who took the WKCE, however, performed better than their MPS counterparts by 6 to 
9 scale score points (.14 to .17 of a standard deviation). Only 61 Choice students in 10 th grade took the 
WKCE, too few to generate reliable performance results. 

This report provides descriptive data regarding the test scores of Milwaukee Parental Choice Program 
students in grades 4, 8 and 10 in reading, math and science, as reported to the School Choice 
Demonstration Project 2006-2007. The tables, graphs, and histograms presented in this report provide 
a snapshot of these students’ percentile scores or scale scores relative to overall national norms or MPS 
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students. They do not and cannot tell us if MPCP schools are performing better, worse, or the same as 
MPS schools. Any reliable determination of the effectiveness of a school choice program like the MPCP 
can only come from a rigorous longitudinal study that follows a representative group of choice students 
over time and compares their achievement gains to those of a comparable set of public school students. 
For such an evaluation, we refer readers to the MPCP Longitudinal Educational Growth Study (LEGS) 
also being conducted by the SCDP 

We do not and cannot present school level information connected to named schools in this or any 
other SCDP report. This restriction is primarily because of requirements concerning the preservation 
of anonymity in academic research. In addition, due to student self-sorting among schools, school 
level information about individually named schools would not be appropriate for evaluating school 
performance or the effects of the Choice program as a whole. 

The comprehensive longitudinal evaluation of the Milwaukee Parental Choice Program, of which this 
report is a part, is being conducted by the School Choice Demonstration Project at the University of 
Arkansas in collaboration with researchers as the University of Wisconsin-Madison and Westat. This 
project is being funded by a diverse set of philanthropies including the Annie E. Casey, Joyce, Kern 
Family, Lynde and Harry Bradley, Robertson, and Walton Family foundations. We thank them for their 
generous support and acknowledge that the actual content of this report is solely the responsibility of 
the authors and does not necessarily reflect any official positions of the various funding organizations, 
the University of Arkansas, the University of Wisconsin, or Westat, Inc. We also express our gratitude 
to MPS, the private schools in the MPCP, and the state Department of Public Instruction for willing 
cooperation, advice, and assistance. 1 



Introduction 

On March 10, 2006, Wisconsin Governor Jim Doyle signed Wisconsin Act 125 into law. The Act 
modified several elements of the Milwaukee Parental Choice Program (MPCP), the nation’s first and 
largest urban school voucher program. Participating private schools are now required to administer a 
nationally normed standardized test annually in reading, mathematics, and science to their MPCP (a.k.a. 
“Choice”) students enrolled in the 4 th , 8 th , and 10 th grades. Beginning in 2006 and extending through 
2011, the individual student results of the tests must be provided to the School Choice Demonstration 
Project (SCDP). Finally, Act 125 requires that: 

The [Wisconsin] legislative audit bureau shall review and analyze the standardized test data 
received from the School Choice Demonstration Project. Based on its review, in 2007 and 



1 We would also like to thank the following advisory board members for their expert guidance regarding this particular 
report: David E. Campbell, University of Notre Dame; Anneliese Dickman, Milwaukee Public Policy Forum; Laura 
Flamilton, RAND; Tom Loveless, The Brookings Institution; Thomas Nechyba, Duke University; and Margaret Raymond, 
The Floover Institution. We are also grateful to the following staff members who were responsible for the enormous 
data entry task regarding this project: Levi Foster, Karen Lockhart, Jacqueline Lockhart, and Kim Torres. Any remaining 
errors are solely the responsibility of the authors. 
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annually thereafter until 2011, the bureau shall report to the legislature under s. 13.172 (2) 
the result of the standardized tests administered under subd. 1., the scores of a representative 
sample of pupils participating in the program under ss. 118.30 and 121.02 (1) (r), and scores of 
a comparable group of pupils enrolled in the school district operating under this chapter on the 
tests under ss 118.30 abd 121.02 (1) (r). 2 

This report describes the results of the student testing conducted by MPCP schools during the 2006-07 
academic year. The standardized test scores were collected from participating schools throughout the 
school year, with most of them arriving at the SCDP in the summer of 2007. The SCDP staff carefully 
entered these scores into a single database and delivered the data to the Legislative Audit Bureau on 
December 28, 2007. 3 

Most of the test scores we received from Choice schools were based on the administration of nationally 
normed tests such as the Terra Nova or the Iowa Test of Basic Skills (ITBS). These types of tests measure 
performance relative to other students by including questions meant to produce a full range of scores 
(i.e. very easy questions ranging to very difficult questions to separate the highest and lowest performing 
students). In contrast, 40 private schools participating in the MPCP administered the Wisconsin state 
criterion referenced test — the Wisconsin Knowledge and Concepts Examinations or WKCE (Table 1). 
Even though the producers of both norm and criterion referenced tests claim that the assessments cover 
the same subject domain, students likely exhibit different patterns of proficiency on the two types of tests 
due to differences in framework and item selection. The WKCE is only administered in Wisconsin; 
therefore, no national distribution exists to make a direct performance comparison with the ITBS and 
Terra Nova. 



To account for these differences, distinct subsections of this report present consolidated results from 
schools that administered the various nationally standardized tests and separate results from schools 
that administered the state-specific exam. For reference purposes, the results of the subgroups of Choice 
students and schools that administered the WKCE are compared with the overall and school-level results 
on the WKCE of economically disadvantaged Milwaukee Public School (MPS) students participating in 
the federal lunch program. 4 



2 Wl Act 1 25, Sec. 8, 1 1 9.23 (7)(e), 2. 

3 The majority of MPCP schools administered the standardized tests late in the spring. The companies that produce the 
tests then require several months to score them and send test result reports to the schools. Since most schools operate 
with a minimum staff over the summer, in some cases the test result reports were not compiled and sent to the SCDP 
until the fall, resulting in an unexpected delay in data input and delivery of the test score database to the Legislative 
Audit Bureau. In an extreme case, the final complete set of 2006-2007 scores from one MPCP school were not received 
until January 2008, requiring the research team to compile and deliver to the LAB a second, updated version of the test 
score database. 

4 As a mechanism for comparing MPCP and MPS students, eligibility for the federal lunch program is limited in two 
ways. First, the family income ceiling for eligibility for the lunch program is 185 percent of the poverty line, which is 
slightly higher than the income ceiling of 1 75 percent of poverty for initial eligibility for the MPCP but somewhat below 
the income ceiling of 220 percent of poverty for renewal of MPCP eligibility. Second, many students who are income- 
eligible for the federal lunch program choose not to participate. The rate of non-participation tends to increase steadily 
as students move from the lower grades to the higher grades. Although federal lunch program participation is an 
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Table 1 . Types of Tests Taken by MPCP Schools and Students 



Schools Students 

Number Percentage Number Percentage 



National Norm 


66 


62% 


5,194 


81% 


WKCE 


40 


38% 


1,231 


19% 


Total 


106 


100% 


6,425 


100% 



The scores received from the MPCP schools are summarized in two results sections below. The first 
results section describes the aggregate scores for the groups of Choice students in each of the tested grades 
who took either nationally normed assessments or the WKCE. The second results section describes the 
distribution of MPCP test scores, by grade and subject matter, at the individual school level. 

Although school-level test scores are presented in the second results section, individual schools are not 
identified by name. Connecting any individual Choice school explicitly to information about its students 
could enable readers to identify individual participants in the study in violation of the assurances of 
confidentiality that are required when conducting such research. 5 The preservation of the anonymity of 
participants in educational evaluations is so important that the federal statute establishing the evaluation 
division of the U.S. Department of Education expressly prohibits the naming of individual students, 
parents, or schools in any of its reports. 6 Because state law requires the affected MPCP schools to 
administer tests and submit scores to the SCDP, we do mention by name in this report the schools that 
did and did not perform those required actions (Appendix A). Because the information thus submitted is 
to facilitate an education evaluation, however, any subsequent presentation of the data provided by schools 
and students must remain anonymous. Of the hundreds of statistical studies of school choice programs 
with which we are familiar, none of them have revealed school-level information directly associated with 
named schools, for these very reasons. 7 



imperfect measure of family disadvantage, it was the best criterion available to generate approximate comparisons 
for this particular element of the evaluation. For this and other reasons described in this report, readers are cautioned 
against drawing any strong conclusions about the relative performance of MPCP and MPS students from the descriptive 
comparisons provided here. 

5 As with all academic research with human participants, the SCDP research team had to gain approval of an extensive 
protocol for protecting the anonymity of participants and the confidentiality of the information that they provide before 
research on the MPCP could begin. Approvals of our research protocols, which prohibit us from associating any data 
with named individuals or schools, were obtained from the Institutional Review Boards for Pluman Subjects Research at 
the University of Arkansas and Westat. 

6 "The Director shall ensure that all individually identifiable information about students, their academic achievements, 
their families, and information with respect to individual schools, shall remain confidential... "See Education Sciences 
Reform Act of 2002, 20 U.S.C., 1 232g, 1 232h. 

7 Many schools in the MPCP provide school-level information by school name voluntarily to parents and organizations 
to facilitate the school choice process. In Milwaukee and in other major cities around the country, the organization 
Great Schools Dot Net is spearheading efforts to consolidate such information into comprehensive school choice guides 
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The MPCP Annual School Testing Summary Report has important strengths and limitations. The main 
strength of the Report is its ability to provide a data-rich snapshot of the current academic performance 
of a large number of students from nearly all of the schools participating in the MPCP Such information 
on the Choice program has not been available for more than a decade. The limitations of the study stem 
largely from the fact that the data are cross-sectional, not longitudinal, and the two comparison groups 
involved are far from ideal. These data show us how well a large group of Choice students is performing 
academically, but tell us nothing about what has caused them to perform at the level. Since many 
factors including the backgrounds and home lives of students as well as the quality of their educational 
experiences likely influence their performance on standardized tests at a given point in time, it would be a 
mistake for readers to draw conclusions concerning the effectiveness of the MPCP based on these simple 
annual descriptive statistics. 

The Longitudinal Educational Growth Study, also being conducted by the SCDP, will entail a comparison 
of the achievement gains over time of a representative sample of MPCP students relative to a carefully- 
matched set of peers in Milwaukee traditional public and charter schools. It will track the performance 
of the same set of MPCP and MPS students as they progress through their educational experience. The 
MPCP Annual School Testing Summary Report, in contrast, examines a different set of MPCP students 
each year at fixed points in their educational experience. It is not a growth or value added comparison 
against peers in MPS or any other group. This point cannot be stressed enough. Readers who are 
interested in student performance differences that can be reasonably attributed to the influence of the 
Choice program itself are advised to follow the progress of the MPCP Longitudinal Educational Growth 
Study. 

Before we present the results of the MPCP student testing for 2006-07, however, we first explain how the 
MPCP test scores were acquired. 



Process for Obtaining MPCP Test Scores 

The School Choice Demonstration Project engaged in an extensive set of activities during the 2006 
academic year that culminated in this report. In August of 2006 Patrick Wolf, principal investigator of 
the SCDP, sent a letter to each school participating in the MPCP informing them of the new testing 
requirements mandated by Wisconsin Act 125 (Exhibit 1). On September 7 th , 2006, Dr. Wolf, along 
with several other members of the SCDP research team, traveled to Milwaukee to attend the first 
Pupil Assignment Council meeting of the 2006-07 school year. During this meeting Dr. Wolf gave a 
presentation to the MPCP school administrators again informing them of the accountability testing 
requirements mandated by Wisconsin Act 125. During the late fall of 2006, Laura Jensen, SCDP 
Research Associate, contacted each MPCP school to inquire about their testing plans for the 2006-07 
school year. In early February 2007, each of the 108 MPCP schools involved in the accountability testing 8 



called "My School Chooser." Because such efforts are voluntary and are not part of a research evaluation, the prohibition 
against connecting descriptive information to named schools does not apply to such school choice guides. 

8 Only participating schools that serve MPCP students in grades 4, 8, or 1 0 are required to administer tests under 

Wisconsin Act 125. Fourteen of the 122 MPCP schools did not enroll any MPCP students in the testing grades in 2006-07 
and therefore were not required to participate in the testing program. 
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was sent a letter informing them of security protocols to be followed when mailing student-level scores to the 
SCDP (Exhibit 2). The February letter also mentioned that scores from fall 2006 testing should be mailed 
to the SCDP by March 1 st , 2007, and that scores from spring 2007 should be mailed to the SCDP by July 1 st ’ 
2007. In late May 2007, a letter was mailed to the MPCP administrators reminding them that all 2006-07 
test scores should be mailed to the SCDP as soon as possible (Exhibit 3). 

On July 1 st , 2007, the SCDP had received 2006-07 test scores from 85 of the 108 affected MPCP schools. 
Through subsequent contacts with the remaining schools, the SCDP soon received test scores from an 
additional 16 schools. By mid-September 2007, the SCDP had obtained test scores from all but seven of 
the MPCP schools required to test. 

On September 22, Dr. Wolf informed officials at the Wisconsin Department of Public Instruction (DPI) 
responsible for overseeing the MPCP of the seven schools that had yet failed to comply with the 2006-07 
testing requirements. The schools were subsequently contacted by the DPI and informed that they 
would not be receiving further voucher payments until they supplied 2006-07 test scores to the SCDP. 9 
Consequently, five of the seven non-complying schools provided the SCDP with 2006-07 test scores for 
their students. By the conclusion of data collection for this report, the SCDP had received test scores 
from 106 of the 108 affected schools, a 98 percent response rate (Appendix A). 



Use of Percentile Rankings 

Because private schools administered several types of tests, as permitted by Act 125, it is necessary to 
use the standardized version of the student scores in order to combine and compare them. Scale scores 
often are the preferred metric for reporting test scores because they are produced by applying a complex 
scoring system, called Item Response Theory (IRT), which draws from the entire pattern of correct and 
incorrect answers and factors in different levels of difficulty. Scale scores are “vertically equated,” meaning 
that scores on the same test can be compared across grades. However, scale scores are not “horizontally 
equated” in any way across different tests. Even for norm-referenced tests such as ITBS and Terra Nova, 
the scale scores for corresponding national percentiles are drastically different. For example, for the 4 th 
grade test, the corresponding scale score for the 50 th percentile on the ITBS is 200. For the 4 th grade Terra 
Nova, the 50 th percentile scale score is 637. Clearly, the scale scores are inappropriate for averaging across 
tests. Therefore, for all nationally-normed standardized tests, the national percentile rank is used in this 
report. All participating schools from which norm referenced scores were retrieved reported the national 
percentile ranking for reading, math, and science, so these scores were consolidated to produce the aggregate 
performance totals provided below. 10 



9 Sarah Carr, "Money Withheld from 7 Schools: Researchers Want Test Scores for Voucher Study," Milwaukee Journal Sentinel, 
September 30, 2007. 

10 Although all of these scores are similar in that they describe the student's performance in comparison with the national 
sample of students that took the test (i.e. the "norming" population), that national sample can vary somewhat across the tests 
- another reason why readers are cautioned against drawing strong conclusions from these illustrative data. 
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In the case of the WKCE, no national norm exists and thus no national percentile exists. Therefore, 
this report presents the school level data in terms of scale scores for the subset of MPCP schools that 
administered the WKCE. Because MPS uses the WKCE, we provide information about the performance 
of income-disadvantaged MPS students to provide a general context in which to interpret the scale 
score performance levels of the MPCP students and schools that administered the WKCE as their 
accountability test. 



Percentile Results at the Program/Grade/Subject Level 



The following tables report the information contained in these data concerning the national percentile 
rank averages for Choice students in grades 4, 8, and 10 in the subjects of reading, math, and science. 
Tables 2 through 4 illustrate these scores by grade. 



Table 2 

Grade 4 National Percentile Summary 
Statistics 



Subject 


Mean 


Std Dev 


Obs 


Min 


Max 


Reading 


34.1 


25.2 


564 


1 


99 


Math 


32.3 


25.0 


572 


1 


99 


Science 


28.0 


22.7 


321 


1 


97 



Table 3 

Grade 8 National Percentile Summary 
Statistics 



Subject 


Mean 


Std Dev 


Obs 


Min 


Max 


Reading 


35.9 


24.6 


407 


1 


99 


Math 


33.2 


26.3 


405 


1 


98 


Science 


28.6 


21.7 


291 


1 


95 



Table 4 

Grade 10 National Percentile Summary 
Statistics 



Subject 


Mean 


Std Dev 


Obs 


Min 


Max 


Reading 


38.4 


25.5 


463 


1 


99 


Math 


34.3 


22.9 


447 


1 


98 


Science 


38.8 


25.8 


446 


1 


97 
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The SCDP received scores produced by 967 4 th grade MPCP students. 11 Fourth grade students in the 
MPCP on average scored around or below the lower-third of the national performance distribution in 
all three subject areas. These results are not surprising. The MPCP is limited to low-income inner-city 
students, a disadvantaged population that typically exhibits test scores well below the national average. 
These particular scores do not indicate how well the program serves these students; they merely describe 
the average performance of an educationally disadvantaged group of students at a particular moment in 
time relative to the average student in the nation, most of whom are not disadvantaged. To interpret the 
results, one would say that the Milwaukee Choice students taking a nationally standardized test averaged 
in the 34 th percentile (scored higher than 34 percent) in reading compared to other 4 th graders nationwide 
taking similar tests. They averaged in the 32 nd national percentile in math and the 28 th national percentile 
in science. 



The MPCP students tested in 8 th grade performed slightly higher relative to the 8 th graders nationwide as 
did the MPCP students in 4 th grade (Table 3). The SCDP received scores generated by 834 8 th graders in 
the MPCP. Choice students in 8 th grade, on average, performed around the lower- third of the national 
distributions in all three subject areas. 

The 546 MPCP students tested in 10 th grade scored the highest relative to nation norms of the three 
grade-cohorts tested (Table 4). The 10 th grade Choice students scored near the 34 th percentile in math, the 
38 th percentile in reading, and the 39 th percentile in science. Although these descriptive statistics appear 
to show academic improvement as choice students mature, these grade-cohorts of students are likely to 
be compositionally different. Readers are cautioned against inferring from these data that the MPCP 
necessarily is responsible for the difference in performance between 10 th grade MPCP students and their 
8 th and 4 th grade counterparts. 



Scale Score Results at the Program/Grade/Subject Level 

The fact that a subset of MPCP students as well as MPS students took the WKCE criterion referenced 
test instead of a norm-referenced test allows us to present those results in a different manner. Income- 
disadvantaged MPS students are likely to be more similar to MPCP students than any national norming 
population, since both groups of students live in the same city and qualify as low income. Still, because 
this approximate match is not very precise, and because the subset of MPCP students who took the 
WKCE is not likely to be representative of all MPCP students, readers are cautioned against drawing 
conclusions about the effects of the Choice program from this limited snapshot comparison. 

Table 5 illustrates the summary statistics for 4 th and 8 th grade MPCP students who took the WKCE in 
the fall of 2006 as well as the statistics from similarly income-disadvantaged students in MPS. Only 



1 1 The total number of student test scores by grade reported here and throughout the text are counts of the number of 
different students for whom at least one test score on a norm-referenced test or the WKCE was provided. That number 
for each grade is higher than the "Observations" numbers reported in each table because we separate out the WKCE 
scale scores from the norm-referenced percentile scores and because most but not all of the students produced test 
scores in all three of the subject areas. 
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61 MPCP 10 th graders took the WKCE last year, making aggregate statistics about that small subgroup 
insufficiently reliable to present here. These descriptive statistics show 4 th graders in the MPCP who 
took the WKCE performing 8 to 13 scale points below the average levels of income-disadvantaged 
MPS 4 th graders. This achievement difference is equal to .16 to .29 of a standard deviation of the MPS 
test score distribution. 12 The 8 th grade scores illustrate the opposite as the scale scores for the MPCP 
students on the WKCE are higher in all three subjects, by 6 to 9 scale score points, than those of income- 
disadvantaged MPS students. This achievement difference at the 8 th grade level is equal to .14 to .17 of a 
standard deviation. For both grades and in all three subjects the differences in the average scale scores of 
the two groups are relatively modest in size. 



Table 5 



MPCP and MPS Summary Statistics for WKCE Scaled Scores Student Level 







Read 


MPCP 

Math Science 


Read 


MPS 

Math 


Science 


Read 


Difference 
Math Science 


4th Grade 




Obs 


338 


338 


338 


4654 


4811 


4673 


- 


- 


- 




Mean 


436 


417 


264 


444 


430 


273 


-8 


-13 


-9 




Std. Dev. 


50 


50 


31 


48 


47 


31 


2 


3 


1 


8th Grade 




Obs 


401 


402 


399 


4834 


4917 


4779 


- 


- 


- 




Mean 


494 


503 


365 


485 


497 


359 


9 


6 


7 




Std. Dev. 


53 


44 


42 


52 


47 


41 


0 


-3 


1 



Performance Distributions at the Individual School Level 

This section presents the distribution of results of the 2006-07 MPCP school testing at the individual 
school level in a series of histograms. The histograms shown illustrate grades 4, 8 and 10 scores in reading, 
math and science. Figure 1 represents 4 th grade, and figure 2 shows 8 th grade, in all three subjects. These 
histograms have a normal distribution overlay, which is depicted by the line in each graph. This normal 
distribution overlay is relative to the empirical data that underlay it. That means it is appropriately scaled 
and has the same mean and standard deviation as the data. These histograms allow for slightly more 
disaggregating among these data. For example, the 4 th grade reading percentile mean reported above was 
34.1, indicating students on average ranked right around the lower one-third of the distribution. Looking 



12 The statistical term "standard deviation" represents an indication of the spread of the data around the average. See 
William Mendenhall, Introduction to Probability and Statistics (Boston, MA: PWS-Kent, 1983). Because standard 
deviation is a measure of distance around the mean, it is commonly used to determine the size of a difference between 
two means. In this case, it is simply a way of describing the point differences between the two types of schools. For 
example, concerning 4th grade math scores, a full standard deviation is 50 points; however, the mean difference 
between MPS and MPCP is about a quarter (.26) of a standard deviation or 13 points. 
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at school averages in the 4 th grade reading histogram, however, there were 18 MPCP schools whose 
Choice students averaged higher than the 50 th percentile in their test score performance. 



These particular histograms use frequency 
counts of the number of schools with average 
percentile scores in each decile. For example, 
in the 4 th grade — National Percentile Reading 
figure, the Choice students in 17 schools 
had an average national percentile score 
in Grade 4 reading between the 20 th and 
30 th percentiles. In general, these graphs 
are intended to indicate the distribution of 
schools that underlay the overall averages 
reported in the previous results section. 




Figure 1 . 4 th Grade National Percentile Rank for MPCP Schools 



Grade 4 National Percentile Reading Histogram 




Grade 4 National Percentile Math Histogram 

15 




0 20 40 60 80 

National Percentile Average Math Score 



Grade 4 National Percentile Science Histogram 
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Figure 2. 8 th Grade National Percentile Rank for MPCP Schools 



Grade 8 National Percentile Reading Histogram Grade 8 National Percentile Math Histogram 





Grade 8 National Percentile Science Histogram 




The histograms of MPCP school-level test score performance presented above generally show 
distributions with what is known as a “positive skew.” The school-level performance averages tend to 
cluster around or below the 40 th percentile, with a small group of much higher performing units at the 
upper tail of the distribution. The positive skew of the distributions lessons somewhat in the upper grades 
and takes more of the shape of a normal bell-shaped curve. The distributions reveal that some school-level 
clusters of MPCP students are performing very well relative to national norms; however, most school- 
level clusters of students performed well below national averages. Because these data merely provide a 
snapshot of school-level groupings of students, we cannot infer from them that some MPCP schools are 
much better performing than most MPCP schools, since the results could be due to higher-performing 
groups of MPCP students gravitating towards particular MPCP schools. In other words, we cannot rule 
out student self-sorting as the cause of the school-level performance distributions presented above. 

Table 6 illustrates the WKCE scale score data aggregated to the school level. Although similar to table 
5, we see how statistics change when test scores are distributed among schools and then averaged at the 
school level. Under almost all circumstances, the averages of subgroup averages (e.g. performance by 
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school) will provide different results than taking the total average of the population (e.g. all testers). 13 
Because a small number of MPCP schools have students performing well above both the MPCP and 
MPS school level averages, the WKCE scores averaged at the school level result in comparisons somewhat 
more favorable to the MPCP schools than the comparisons based on test-scores at the individual student 
level. The performance of school-level groupings of 4 th grade Choice students is the same as, or very 
slightly lower than, the performance of school-level groupings of 4 th grade income-disadvantaged MPS 
students. A similar comparison of school-level results in 8 th grade shows MPCP school averages that are 
higher than those of 8 th grade income-disadvantaged MPS students by 21 to 27 scale score points (.62 to 
.82 of a standard deviation). 



Table 6 



MPCP and MPS Summary Statistics for WKCE Scaled Scores School Level 





Read 


MPCP 

Math Science 


Read 


MPS 

Math 


Science 


Read 


Difference 
Math Science 


4th Grade 




Obs 26 


26 


26 


129 


130 


128 


- 


- 


- 




Mean 442 


423 


270 


442 


428 


272 


0 


-5 


-2 




Std. Dev. 37 


43 


26 


30 


29 


19 


7 


14 


7 


8th Grade 




Obs 36 


36 


36 


98 


98 


98 


- 


- 


- 




Mean 509 


512 


376 


481 


491 


353 


27 


21 


22 




Std. Dev. 33 


30 


36 


29 


27 


30 


4 


4 


7 



The following 12 histograms graphically show the Table 6 data for those MPCP schools and the MPS 
schools taking the WKCE. 



Grade 4 Choice WKCE Reading Histogram Grade 4 MPS WKCE Reading Histogram 





13 In statistical parlance, the fact that an average of subgroup averages rarely equals the average of the entire group is 
called the "ecological inference" phenomenon. 
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As evidenced in the above two graphs, the means of the two populations are almost identical, but clearly 
the MPCP group has a higher standard deviation (e.g. greater variation in performance) and has more 
schools in the upper tail. We would expect a wider variation in school-level achievement among the 
variegated set of MPCP schools. 



Grade 4 Choice WKCE Math Histogram Grade 4 MPS WKCE Math Histogram 





A similar story is true for the math scores; however, the standard deviation is greatest for this 4 th grade 
subject. The mean difference is also greatest between these two groups with MPS schools averaging 5 
points higher than the MPCP schools. 



Grade 4 Choice WKCE Science Histogram Grade 4 MPS WKCE Science Histogram 





The science scores for these two groups appear to be more normally distributed than the other two 
subjects. However, the standard deviation is again higher for the set of MPCP schools. 
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Grade 8 Choice WKCE Reading Histogram Grade 8 MPS WKCE Reading Histogram 





In the case of 8 th grade reading, we see a relatively normal distribution for each of the two groups. 

The MPCP histogram shows a higher mean than the MPS figure, by 21 points, but there are more 
schools represented in the lower tail of the MPCP distribution than in the upper tail. The school-level 
distributions for math are roughly similar to those for reading. 

Grade 8 Choice WKCE Math Histogram Grade 8 MPS WKCE Math Histogram 





Grade 8 Choice WKCE Science Histogram 



Grade 8 MPS WKCE Science Histogram 





The science histograms for 8 th grade also show relatively normal distributions for each group. The 
8 th grade science mean in both groups has been influenced by a spike at that mean. For the MPCP 
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distribution, it appears more schools are falling in the upper tail; whereas, more schools seem to fall in the 
lower tail in the MPS distribution. 




These tables and histograms suggest there are schools in each group (Choice and MPS) in which students 
are performing well above or well below the city average. These diagrams suggest there is ample variation 
in both groups. The standard deviations for the MPCP group are higher, which is likely because of 
the lower number of schools and 



greater diversity of schools in that 
group. One should be careful 
in interpreting these data. The 
differences across the comparison 
groups and between the grades 
cannot necessarily be attributed 
to the Choice program. The 
figures are presented here merely 
to provide a general description 
of the school-level performance 
of MPCP students and place that 
performance in context by making 
a rough comparison with the 
school-level performance of income 
disadvantaged MPS students. 



Summary 

The purpose of this report is to provide descriptive data regarding the test scores of Milwaukee Parental 
Choice Program students in grades 4, 8 and 10 in reading, math and science, as reported to the School 
Choice Demonstration Project 2006-2007. The above tables, graphs, and histograms provide a snapshot 
of these students’ percentile scores relative to overall national norms, and scale scores on the WKCE 
relative to income-disadvantaged MPS students. Because national norms are based on students with 
“average” educational circumstances, and the MPCP exclusively serves low-income inner-city students, the 
fact that their average levels of performance on norm referenced tests cluster around the lower one-third 
of the distribution is not surprising and should not be interpreted to indicate that the MPCP in general 
or MPCP schools in particular are doing a poor job of educating students. The comparison of the scale 
scores of the Choice students and schools that took the WKCE with the scores of income-disadvantaged 
MPS students and schools similarly is presented descriptively with no claim that the schools themselves 
were independently responsible for the various results. Any reliable determination of the effectiveness of 
a school choice program like the MPCP can only come from a rigorous longitudinal study that follows 
a representative group of choice students over time and compares their achievement gains to those of a 
comparable set of public school students. The MPCP Longitudinal Education Growth Study (LEGS) 
will serve as the proper source for such a determination. 
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SCHOOL CHOICE 
DEMONSTRATION PROJECT 

GEORGETOWN UNIVERSITY 



UNIVERSITOARKANSAS 



=■!■»»= 



School Choice Demonstration Project 
3300 Whitehaven Street NW, Suite 5000 
Washington, DC 20007 
(202) 687-6323 
Fax: (202) 687-1515 



College of Education and Health Professions 



Department of Education Reform 
201 Graduate Education Building 



Fayetteville, Arkansas 72701 
(479) 575-3172 
Fax: (479) 575-3196 



August 14, 2006 

Administrator Name 
«School» 

Street Address 
City, State ZIP 

Dear <School Administrator^ 

My name is Patrick Wolf. I lead a research team called the School Choice Demonstration Project 
(SCDP). I am writing to urge you to attend an informational meeting regarding our upcoming 
evaluation of the Milwaukee Parental Choice Program (MPCP). The session will take place on 
September 7 th at 5 pm at Corpus Christi School, 8545 West Villard Avenue, immediately after the 
Pupil Assignment Council meeting. 

The evaluation of the MPCP is a new undertaking required by state law. Last March, the 
Legislature passed and the governor signed Wisconsin Act 125, which raised the student cap on 
the program by 50 percent and called for an evaluation of the program by the SCDP. Private 
schools participating in the MPCP are now required to: 

1. Administer annually a nationally normed standardized test of their choosing in reading, 
mathematics, and science to all MPCP students in their school in grades 4, 8, and 10; 

2. Forward the scores from all standardized tests they administer to the SCDP for analysis. 

I want to assure you that your school’s test scores will be kept confidential. Test scores will NOT 
be reported to anyone by student or by individual school. They will only be used to evaluate the 
performance of the MPCP as a whole and general types of participating schools (e.g. Catholic, 
Lutheran, independent). 
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To provide a more rigorous evaluation of the program, we plan to test a representative sample, 
or “panel,” of MPCP students in grades 3-10 using the WKCE. Our goal is for those students 
to be tested under conditions that are as similar as possible to those experienced by comparable 
MPS students when they complete the WKCE. We aim to test the students in the MPCP 
panel during a regular school day near the end of November in their own schools using school 
personnel if possible. To do so we will need your help. 



The purpose of the meeting on September 7 th is for us to discuss in greater detail our plan for 
handling the evaluation requirements in the new MPCP law and to consider your questions and 
concerns. Several members of the research team and officials from the Department of Public 
Instruction will be in attendance. We view it as a great opportunity for a productive dialogue 
regarding this new requirement of the MPCP. 



If you have any questions regarding the session, please call me at 479-575-2084 or James Rahn, 
SCDP Senior Research Associate, at 414-443-8971. I hope to meet you on the 7 th . 



Sincerely yours, 



Patrick J. Wolf, Ph.D. 

Professor and Endowed Chair in School Choice 
Principal Investigator, SCDP 



Cc: Lisa Geraghty, Department of Public Instruction 

Tricia Collins, Department of Public Instruction 
James Rahn, Wisconsin Lutheran College and SCDP 



Milwaukee Longitudinal School Choice Evaluation: Annual School Testing Summary Report 



Exhibit 2 



18 



February 2008 




/ \ 



SCHOOL CHOICE 
DEMONSTRATION PROJECT 

GEORGETOWN UNIVERSITY 



School Choice Demonstration Project 
3300 Whitehaven Street NW, Suite 5000 
Washington, DC 20007 
(202) 687-6323 
Fax: (202) 687-1515 



February 6th, 2007 




UNIVERSITY^ARKANSAS 

=IMI= a 

College of Education and Health Professions 
Department of Education Reform 
201 Graduate Education Building 
Fayetteville, Arkansas 72701 
(479) 575-3172 
Fax: (479) 575-3196 



Administrator Name 
«School» 

Street Address 
City, State ZIP 

Dear <School Administrator^ 

As our previous correspondence mentioned, one of the major components of the evaluation of the 
Milwaukee Parental Choice Program (MPCP) mandated by Act 125 requires that all participating schools 
test their choice students in grades 4, 8, and 10 annually in reading and math and, for grades 4 and 8, 
also in science. The Act also requires MPCP schools to submit the scores from all standardized tests 
they administer to the School Choice Demonstration Project (SCDP) at the University of Arkansas. The 
purpose of this letter is to provide you with guidance regarding how to fulfill these new requirements. 

Time Frame & Mailing Address 

Results of any fall testing that was administered at your school should be mailed to the 
1 st . Results of any spring testing that your school administered should be mailed to the 
Test score information should be mailed to the following address: 

Laura Jensen 

School Choice Demonstration Project 
Department of Education Reform 
20 1 Graduate Education Building 
Fayetteville, AR 72701 
479-575-6345 

Format of Scores 

A copy of the scores from any standardized test (e.g. Terra Nova, ITBS, WKCE, etc.) administered at 
your school can be submitted to the SCDP in either electronic or paper format, though electronic format 
is strongly preferred. 



SCDP by March 
SCDP by July 1 st . 
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Security Protocols 

To protect the confidentiality of your students, it is important certain data protection strategies 
be implemented. If you are sending an electronic copy of your students’ scores, you must 
password protect the file and bum the file to a CD (see attached sheet with instructions on CD 
burning). Instructions on how to password protect Microsoft Word, Microsoft Excel, and Acrobat 
Professional documents can be found below. 

Securing a Word or Excel file with a password: 

With the file open: 

• On the Tools menu, click Options, and then click Security. 

• In the Password to open box, type MPCP2006, and then click OK. 

• In the Reenter password to open box, type MPCP2006 again, and then click OK. 

Securing an Acrobat Professional/PDF file with a password: 

With the PDF document open: 

• Control D for Document Properties 

• Click Security Tab > Change Security Method to Password Security 

• Check “Require a password to open the document” 

• Enter a password MPCP2006 in the “Document Open Password:” box 

• Confirm the password MPCP2006 

If the electronic file containing your students’ scores is not in any of these formats, please check 
the help menu in the program that fits the file for guidance in how to password protect the file. 
Another option would be to convert the file from its program format into Excel, which is the most 
preferred format for transferring data files securely. 

If you chose to submit a paper copy of your students’ test scores, the package in which your 
documents are mailed must be sealed with the mailer’s signature over a taped seal. When mailing 
your scores, make sure that you indicate that a signature of the addressee is required at the point 
of destination (i.e. School Choice Demonstration Project). Paper copies of scores can be mailed 
securely by Registered U.S. Mail, Federal Express, UPS, etc. 

We look forward to receiving your school’s test scores. As always, if you have any questions, 
please call Laura Jensen, SCDP Research Assistant, at 479-575-6345. 

Sincerely yours, 



Patrick J. Wolf, Ph.D. 

Professor and Endowed Chair in School Choice 
Principal Investigator, SCDP 
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SCHOOL CHOICE 
DEMONSTRATION PROJECT 

GEORGETOWN UNIVERSITY 



College of Education and Health Professions 



UNIVERSITY^ARKANSAS 



School Choice Demonstration Project 
3300 Whitehaven Street NW, Suite 5000 
Washington, DC 20007 
(202) 687-6323 
Fax: (202) 687-1515 



Department of Education Reform 
201 Graduate Education Building 



Fayetteville, Arkansas 72701 
(479) 575-3172 
Fax: (479) 575-3196 



May 24 th , 2007 

Administrator Name 
«School» 

Street Address 
City, State ZIP 

Dear <School Administrator^ 

I hope this letter finds you doing well as you are wrapping up the end of what I trust was a successful 
2006-07 academic year. 1 want to begin by thanking you for all your assistance throughout this year as 
we gathered data for the baseline year of the Longitudinal Evaluation of the Milwaukee Parental Choice 
Program. We are so pleased with the quick turnaround response we have received each time we have asked 
for information from you. The participation by the MPCP schools has been exceptional. 

There are still some schools that have not mailed us their principal surveys or their fall and/or spring 
standardized test scores. As a reminder we need a copy of all individual-level test results from ANY 
standardized testing that your school did during this school year. Please send any outstanding documents to 
the following address as soon as possible: 

Laura Jensen 

School Choice Demonstration Project 
University of Arkansas 
20 1 Graduate Education Building 
Fayetteville, AR 72701 

As always, if you have any questions, please do not hesitate to give Laura Jensen a call (479-575-6345). I 
wish you the best of luck with your end of the school year activities. 

Sincerely, 

Patrick Wolf, Ph.D. 

Principal Investigator 

School Choice Demonstration Project 
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Atlas Preparatory Academy 

Atonement Lutheran School 

Believers in Christ Christian 
Academy 

Bessie M. Gray Prep Academy 

Blessed Sacrament School 

Blyden Delany Academy 

Catholic East Elementary School 

CEO Leadership Academy 

Ceria M. Travis Academy, Inc. 

Christ Memorial Lutheran 
School 

Christ St. Peter Lutheran School 

Christian Faith Academy of 
Higher Learning 

Clara Mohammed School 

Community Vision Academy 
LTD 

Concordia University School and 
Institute for LIGHT 

Corpus Christi School 

Destiny High School 

Divine Savior Holy Angels High 
School 

Dr. Brenda Noach Choice 
School 

Early View Academy of 
Excellence 

Eastbrook Academy 

Emmaus Lutheran School 

Excel Academy 



Excel Learning Academy 

Fairview Lutheran School 

Family Montessori School 

Garden Homes Lutheran School 

Gospel Lutheran School 

Greater Holy Temple Christian 
Center 

Harambee Community School 

Hickman Academy Preparatory 
School 

Holy Redeemer Christian 
Academy 

Holy Wisdom Academy 

Hope Christian School 

Hope Middle School 

Institute for Career 
Empowerment Inc. 

Jared C. Bruce Academy 

Johnson Christian Academy, Inc. 

Keal Preparatory School, Inc. 

King’s Academy Christian 
School 

LaBrew Troopers Military 
University School 

Lutheran Special School & 
Education Services 

Marquette University High 
School 

Mary Queen of Martyrs 
Messmer High School 
Messmer Prep Catholic School 



Milwaukee Lutheran High 
School 

Milwaukee Montessori School 

Milwaukee Seventh Day 
Adventist School 

Mother of Good Counsel School 

Mount Calvary Lutheran School 

Mount Lebanon Lutheran 

New Testament Christian 
Academy 

Noah’s Ark Preparatory 

Northwest Lutheran School 

Notre Dame Middle School 

Nzingha Institute of Creative 
Learning for Living 

Oklahoma Avenue Lutheran 
School 

Our Lady of Good Hope School 

Our Lady of Sorrows School 

Our Lady Queen of Peace Parish 

Parklawn Christian Leadership 
Academy 

Pius XI High School 
Prince of Peace 

Resurrection Christian Academy 
Risen Savior Lutheran School 
Saint Adalbert School 
Saint Anthony School 
Saint Bernadette School 
Saint Catherine of Alexandria 
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Saint Catherine School 

Saint Charles Borromeo School 

Saint Gregory the Great Parish 
School 

Saint Joan Antida High School 

Saint John Kanty School 

Saint John’s Evangelical 
Lutheran 

Saint Josaphat Parish School 

Saint Leo Catholic Urban 
Academy 

Saint Marcus Lutheran School 

Saint Margaret Mary School 

Saint Martini Lutheran School 

Saint Peter-Immanuel Lutheran 
School 

Saint Philip Neri Catholic 
School 



Saint Philip’s Lutheran School 

Saint Rafael the Archangel 
School 

Saint Roman Parish School 

Saint Rose Catholic Urban 
Academy 

Saint Sebastian School 

Saint Thomas Aquinas Academy 

Saint Vincent Pallotti School 

Salam School 

Sharon Junior Academy 

Siloah Lutheran School 

Tamarack Community School 

Teenpreneur #2 

Texas Bufkin Academy 

The Hope School 

Travis Technology High School 



MPCP Schools with no MPCP Students in Tested Grades 2006-2007 
N=14 



Agape Center of Academic 
Excellence, Inc. 

Carter’s Christian Academy 

CrossTrainers Academy 

Pamily Academy 

First Steps Christian Learning 
Academy 

Grace Preparatory School of 
Excellence 

Grandview High School (Seeds 
of Health) 

Kindergarten Plus 

Malaika Early Learning Center 



Paige II University School, Inc. 

Parkside Elementary (Seeds of 
Health) 

Sherman Park Lutheran School/ 
Preschool 

The AppleCrest Preparatory 
Leadership Academy 

Windlake Elementary (Seeds of 
Health) 



Trinity Christian Academy for 
Nonviolence 

Urban Day School 

Veritas Academy 

Victory Christian Academy 

Victory Preparatory Academy 

Washington DuBois Christian 
Leadership Academy 

Wisconsin Lutheran High 
School 

Word of Life Evangelical 
Lutheran School 

Yeshiva Elementary School 

Young Minds Preparatory School 
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